Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a ``periodic inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the $\relu$-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

Weaknesses: A significant shortcoming in the approach is the lack of proper and thorough validation with recurrent neural networks. The stated problem (i.i.d. in fourier space, restricted to a compact region in R d) can be tackled with the auto-regressive approach provided by RNNs. However this is not mentioned, or compared against in the paper. I note that RNNs are somewhat compared against in the appendix, but this is with respect to RNN Snake vs RNN. Above all this paper needs a comparison showing the ability of feedforward Snake networks to outperform vanilla RNNs.

I think this is an interesting submission, that lead to a detailed discussion among the reviewers. Overall the work is novel and looks at an interesting question regarding extrapolation (and dealing with periodic functions). Overall I agree with some of the reviewers that the motivation and generally the write-up of the work could be improved, but I think there is already value in the work. I would like to highlight a few points that are worth considering: * a further discussion regarding how the proposed approach compares to RNNs or autoregressive models when it comes to modeling periodicity * there are some concerns regarding the methodology used (e.g.

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, x \sin 2(x), which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the \relu -based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

Sociology professor Anton Oleinik argues that neural networks are structured in a way that limits the possibility that they will ever have true artificial creativity. Neural networks–a common type of artificial intelligence–are infiltrating every aspect of our lives, powering the internet-connected devices in our homes, the algorithms that dictate what we see online, and even the computational systems in our cars. But according to an article published in the peer-reviewed journal Big Data & Society by Anton Oleinik, a sociology professor at Memorial University of Newfoundland, there's one crucial area where neural networks do not outperform humans: creativity. Researchers have projected that automation may claim 800 million jobs around the world by 2030. Others suggest that as many as half of American jobs may be under threat from automation. But amid all the handwringing about robots taking people's jobs, Oleinik's analysis is further evidence that AI will likely only replace repetitive tasks that humans aren't particularly skilled at to begin with.